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CAMPBELL, Calif .— (BUSINESS WIRE)--Aug. 15, 2000 

Portera, a leading vertical service provider delivering innovative 
outsourcing solutions to the professional services industry, today 
announced a partnership with LearningAction, a leading provider of 
Web-based corporate compliance and compliance training solutions. 

LearningAction joins other industry leaders like DigitalThink 
(Nasdaq: DTHK) as content partners for Portera' s ServicePort, the 
f irst-of-its-kind Web-hosted professional services automation (PSA) 
solution with integrated learning management. 

"Especially in the professional services industry, where people are 
the primary asset, employee development is essential, " said Elise Olding, 
vice president of knowledge, e-learning and collaboration at Hurwitz Group, 
a software industry analyst and consulting firm. "Portera has recognized 
this and developed a robust learning solution that is nicely integrated 
into their PSA solution. The integration of learning management in 
ServicePort enables service organizations to be more strategic in their 
employee development, by matching training to pipeline opportunities and 
creating employee-specific learning tracks." 

Through this partnership, Portera is able to provide its ServicePort 
subscribers with industry-leading online compliance training to supplement 
their internally developed courses. LearningAction helps Portera customers 
effectively communicate corporate policies through a combination of online 
legal and compliance training courses. 

"Our partnership with Portera expands our ability to serve the. unique 
needs of the rapidly growing professional services industry, " said 
LearningAction president Josh Abramowitz. "Portera f s 100 percent Web-hosted 
approach and integration with PSA are setting the standard for online 
training in the professional service industry. We are proud to make our 
online legal and compliance training available through Portera 
ServicePort . " 

Gary Steele, CEO of Portera, said, "Developing and retaining talent 
is a crucial part of running a professional service business . That's why 
we've been offering learning management solutions to our customers for 
some time. Portera was the first in the industry to provide a PSA solution 
with integrated learning management and we continue to enhance this unique 
Web-hosted service." 

Portera f s Learning Manager, which was released in February, is a 
comprehensive Web-hosted employee development and training solution 
designed specifically for professional services organizations. It 
simplifies the scheduling, enrollment, payment and tracking of employee 
training and, through integration with ServicePort, enables professional 
service organizations to match training needs to business opportunities 
and measure the return on training investment. 

About LearningAction 

LearningAction, Inc, is the first company devoted to providing 
Web-based training on corporate legal topics. LearningAction systems enable 
a company to train workers on harassment, discrimination, wrongful 
termination, and other legal matters. With LearningAction, a company can 
reach its entire workforce, and can document that workers have participated 
in the training. LearningAction systems are also highly cost-effective, and 
are customizable to suit the needs of different organizations. Learning 
Action is a privately held company based in Boston, Massachusetts. 

About Portera 



A 

' Portera is a leading vertical service provider delivering innovative 
outsourcing solutions to the professional services industry. Professional 
service organizations worldwide leverage Portera' s technology-based 
services to enhance productivity and margins. Leading organizations that 
have chosen to focus on their core business of serving clients by using 
Portera 's solutions include: Agilent, British Telecom, Commerce One, 
E.piphany, Hewlett-Packard, Inventa, Microsoft, Network Associates, Tanning 
Technology and Viant . 

Further information about Portera can be found at www.portera.com. 
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BELLEVUE, Wash .--( BUSINESS WIRE) --Feb . 9, 2000 

click21earn.com, inc. (Nasdaq : CLKS ) , a Wired World Company (TM), today 
announced the immediate availability of Ingenium 5.0, a major upgrade to 
its leading enterprise learning management system. 

The new release of the skills and competency based system enables 
organizations to manage every type of training, from instructor-led through 
technology-based, using one enterprise-wide -solution. Ingenium 's Web 
Connect module allows users to access Ingenium 1 s many features through a 
browser, such as the new training gap analysis, and improved skills and 
competency gap analysis. Ingenium also tracks results for e-^Learning 
content from many vendors, including titles from SmartForce (Nasdaq : SMTF) , 
SkillSoft (Nasdaq: SKIL) , NETg (NYSE : H) , content created in 

click21earn . com 1 s ToolBook authoring suite; and Macromedia's (Nasdaq : MACR) 
Authorware, in addition to AlCC-compliant courseware. This offers learners 
and administrators a seamless, integrated environment for offering and 
managing the optimal mix of e-Learning content, as well as managing 
instructor-led classes. 

&uot; Leading companies make major investments in improving the skills 
and competencies of their employees and need a solution to manage every 
form of learning -- no matter what the delivery vehicle is,&uot; said Kevin 
Oakes, president and CEO of click21earn.com, inc. &uot/0ur Ingenium 
Learning Management System has been the long-time leader in skills 
management, and now fully integrates e-Learning for simplified, 
comprehensive access by learners, managers, and training 
administrators . &uot ; 

iT Ingenium 5.0 extends its skills and competency management strength by 

/introducing graphical &uot; learning maps,&uot; as well as a new training 
/gap analysis feature. Learning maps enable trainers and administrators to 
j assemble learning experiences into a graphical display to guide the learner 
through a series of courses. These maps can be developed for jobs, 
illustrating the specific skills needed to advance to a new position, or 
for organizations, providing the particular certifications or knowledge 
needed as employees move into a new area. The maps can also. be prescribed 
for a specific student based on his/her career development needs . The new 
training gap analysis is a related feature that compares a student's 
completed training with the learning requirements for any combination of 
job and organization. Ingenium' s introduction of the learning map concept 
1 is an important innovation that helps align the development of individual 



'I learners with organizational objectives. 

^ &uot; Ingenium has always provided learning management based on strong 

skills and competency management , &uot ; observes Brandon Hall, Ph.D., editor 
brandon-hall.com, &uot;and the newest version demonstrates their 
understanding of the importance of managing every form of learning from one 
system. Enterprise learning management systems continue to be a 
mission-critical decision for organizations and those pursuing such a 
system should consider this strong new offering from click21earn . com . &uot ; 

Buckman Laboratories, a $310 million privately held specialty 
chemical company, is well-known for its commitment to knowledge management 
innovation and ongoing growth of its employee associates world-wide. They 
have used Ingenium since 1997 and helped shape the expanded features of 
version 5.0. &uot;We needed to simultaneously deliver the same high quality 
product training to all of our associates, any where in the world, whether 
in Indonesia or here locally in Memphis, &uot ; says Sheldon Ellis, Vice 
President of the Bulab Learning Center. &uot; Because Ingenium is 
skills-based, supervisors can sit down with their staff one-on-one ; and map 
out a career path based on the specific skills that will make each 
associate successful. This career mapping approach motivates the individual 
to take responsibility for their own career development . &uot ; Ingenium 1 s 
ability to offer learners a catalog of every type of training, and to 
maintain a complete record of training outcomes, is critical to Buckman' s 
strategy. &uot;We find that e-Learning is a great way to get learners to a 
common knowledge level, making time in the classroom more ef f ective, &uot ; 
adds Ellis. 

Substantial enhancements were made to each of Ingenium' s components. 
These enhancements include: 

Ingenium Web Connect, which allows learners and managers complete 
access to relevant features through a web browser 
-- Ingenium Messenger Express, which uses push technology to 
automate notification of learners concerning registration, 
certification reminders, and event cancellations 
— Ingenium Report Browser, which now includes over 80 
professionally designed Seagate Software Crystal Reports, 
allowing users to export information in many common formats 
including HTML 

-- The newly released Data Update and Import Tool, which simplifies 
integration with existing enterprise applications such as HRIS 
and ERP systems 

Ingenium 5.0 is available in an Enterprise edition appropriate for 
multi-site organizations o'f any size. Smaller organizations can implement 
the lower cost Workgroup edition. Ingenium 5.0 is available immediately 
from click21earn . com' s direct sales organization. 

About click21earn.com, inc. 

click21earn.com, inc. (Nasdaq: CLKS ) formerly known as Asymetrix 
Learning Systems, Inc. (Nasdaq: ASYM) is a leading provider of e-Learning 
solutions to businesses, government, and educational institutions 
throughout the world. By developing a powerful and full-featured ASP 
platform for e-Learning, and partnering with market-leading content 
publishers, click21earn.com is' able to provide customers with instant 
access to their e-Learning Network. Customers can access thousands of 
courses, and use browser-based publishing and learning management tools 
using a custom-configured e-Learning site or by visiting the e-Learning 
Portal at www.click21earn.com. click21earn . com 1 s enterprise e-Learning 
initiatives include learning management products (spearheaded by the market 
leading Ingenium total learning management solution) and web-based 
authoring software products (featuring the acclaimed ToolBook II Instructor 
and Assistant authoring tools) . Each solution is augmented by professional 
services including award winning custom content development, strategic 
consulting and integration services. For more information on 
click21earn. com' s e-Learning solutions, call 800/448-6543 or 4 25/4 62-0501. 
click21earn.com, inc. is located on the World Wide Web at 
www.click21earn.com. 

click21earn.com, 2MinuteTutors , Ingenium, ToolBook II Instructor and 
ToolBook II Assistant are trademarks of click21earn.com, inc. All other 
company and/or product names are the property of their respective owners. • 
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Introduction 

Today, more than ever, training programmes are being held 
accountable for providing the "work-ready" graduates that 
employers want. Yet, after years of research and debate, 
there is still no consensus on the best course of action for 
achieving programme accountability. However, many see 
cause for optimism in an Instructional Systems Methodology 
for planning, developing, implementing and evaluating train- 
ing programmes. This competency (performance)-based 
systems approach ensures that trainees are taught the 
knowledge and skills essential for successful job 
performance. 

Accountability, the driving force behind competency-based 
training, is substantially increased when the principles and 
processes of this carefully conceived instructional system 
are used. This is because the derivative and iterative 
character of the methodology ensures, when its procedures 
are resolutely carried out. that instruction will be job relevant. 

The procedural model for this instructional system com- 
prises five phases: (1) analyse; (2) design; (3) develop; (4) 



implement, and <5) control. Figure 1 provides a graphic por- 
trayal of the interrelationships between the five phases and 
17 procedural steps which form the closed-loop system. The 
model is based on both research and years of practical ex- 
perience in its application. 

Certain components, such as job analysis and learning ob- 
jectives, have been used by trainers and educators for some 
time. Nevertheless, the application was often haphazard and 
fragmentary. This model brings these and other components 
together in a coherent whole. The critical distinction with 
this procedural model is the emphasis on an integrated in- 
structional system. 

Each of the following five headings is devoted to presen- 
ting descriptive information on a phase of the model. The 
model and text can be used in much the same way as a 
cookbook recipe— do this, then that and you have a cake. 

The Main Phases 
Analyse 

The first procedural step in the model is to analyse the job 
under consideration to determine the human performance 
activities. Job analysis is the foundation for sound training 
programmes and the basis for determining the job-relevant 
behaviours that trainees must be capable of performing 
when they complete their instruction. 

Unless personally familiar with the job, a document study 
should be conducted at the outset. This is done to find out 
as much as possible about the job and any job analysis work 
which is available. Once the literature has been reviewed, 
one or more of the job analysis techniques is selected for 
use. The technique selected will depend on its relative utili- 
ty and whether the purpose is to identify and describe the 
tasks and elements of tasks that make up the job or to verify 
existing job analysis information. 

The more common techniques are: 

(a) Observations of expert worker performance; 

(b) Group or individual interviews with expert workers 
and their supervisors; 

(c) Questionnaire surveys of expert workers and their 
supervisors, and 

id) A jury of expert workers and supervisors. 

While the observation and interview techniques are the most 
dependable for gathering detailed job information, no one 
method of job analysis is the best for all situations. In fact, 
the use of a combination of data-gathering techniques, such 
as observation and interviews, is preferable. When several 
of the techniques offer reasonable alternatives, the advan- 
tages and limitations of each should be considered in the 
light of constraints such as time and the availability of pro- 
fessional help (see Table I). 

No matter how well the remaining procedural steps are car- 
ried out. if the job analysis information collected from this 
pivotal step is not valid and reliable, the training programme 
will fail to produce graduates who can perform successful- 
ly on the job. 

Some of the tasks identified can be performed without train- 
ing or seldom will be required on the job. Only minimum 
job degradation would result if such tasks were performed 
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Figure 1 . Instructional Systems Model 
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inadequately by a job-entry worker. Other tasks may be 
highly critical to successful job performance, and their com- 
plex nature or frequency of performance makes instruction 
essental. The needs of the job, economics, time, facilities 
and other considerations require a "best judgment" deci- 
sion when determining which tasks to include in the train- 
ing programme^]. Rejected tasks are not reconsidered until 
the external evaluation step which is performed after a train- 
ing cycle is completed and graduates are on the job. 

In order to avoid unnecessary duplication of effort, existing 
course documentation and other available material should 
be reviewed. This endeavour may reveal that all or portions 
of the instructional system have already been done by 
someone else and are suitable for use. It would not be cost- 
effective, however, to use materials from another source 
which have not been validated or are essentially extraneous. 

Design 

The development of objectives is the beginning step in 
designing training to meet job-performance requirements. 
From this point, instructional design and development is car- 
ried out with respect to learning objectives rather than job 
tasks. Some of the benefits of properly stated objectives are: 



(a) A cJear-cut description of the knowledge and skills 
that are to be attained by the trainee; 

(b) A sound basis for writing test items, and 

(c) Determination of the most suitable instructional 
strategy. 

learning objectives provide the behavioural actions, perfor- 
mance conditions and attainment standards which can be 
measured within the instructional setting (see Table II for 
guidelines in writing the three parts of a learning objective). 

Terminal learning objectives are those formulated for the 
tasks that were selected for instruction. Each germinal lear- 
ning objective is subjected to a hierarchical analysis to derive 
subordinate or enabling objectives. These enabling objec- 
tives provide the level of detail necessary to specify the 
behavioural elements, and the skills and knowledge that help 
the trainee achieve a terminal objective[2j. In many cases, 
the resultant enabling objectives will specify the same 
behavioural actions as the elements of the job-task itself. 

Learning objectives bridge the gap between task perfor- 
mance under norma! work conditions and learning how to 
perform that task in an instructional setting. They serve as 
a control over the content and output of instruction. The 
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Table I. Advantages and Limitations of 
Selected Job-analysis Techniques 



Technique 



Advantages 



Limitations 



Observation of ex- 
pert workers doing 
the job at the work 

site 



Questionnaire sent 
to expert workers 
and their supervisors 



Jury of experts 



Finds out what in- 
cumbents actually 
do, not what they 
say they do 

Accomplished 
directly by 
observer or in- 
directly with 
cameras 

Few people 
involved 



In-person interviews 
with expert workers 
and their supervisors 



High response rate 
from selected 
incumbents 

Yields high-quality 
information 



Telephone interviews 
with expert workers 
and their supervisors 



High response rate 
from selected 
incumbents 

Inexpensive 

Information col- 
lected in a short 
time 

Can survey a large 
random sample of 
incufmbents 
economically by 
mail 

Yields large 
amounts of 
information 
Responses are 
adaptable to 
machine tabulation 
and statistical 
analysis 

Inexpensive and 
reliable 

Information col- 
lected and verified 
in a short time 
Few people 
involved 



Can only observe a 
sample of all the 
job tasks 

Expensive and 
time-consuming 

Direct observation 
can bias worker 
performance 

Requires a trained 
observer 



Expensive and time- 
consuming to con- 
duct interviews and 
compile data 
Results depend on 
the co-operation of 
those interviewed 

Interferes with 

operational 

activities 

Requires a trained 
interviewer with a 
pleasant personality 
Less sensitive than 
in-person interviews 
Requires a trained 
interviewer with a 
good telephone 
manner 

Requires know-how 
to prepare a ques- 
tionnaire with ex- 
plicit instructions 
Places a heavy de- 
mand on recall by 
those completing ft 

Low response rate 
(especially if filled 
out In a person's 
own time) 

May be incomplete 
or filled out by an 
unqualified person 
Requires time away 
from the job 

Requires trained in- 
terviewer and 
recorder 



ability to develop objectives (teach and test those objectives) 
is the foundation for trainee achievement. 

After the learning objectives have been prepared, test items 
to measure their attainment are written. These test items, 
called criterion-referenced measures, test the required 
behavioural actions under the conditions and standards 
specified in the objectives.* For contrast, a norm-referenced 



Table II. Writing the Three Parts of 
a Learning Objective 



Behavioural actions 
(what trainee does) 



Performance 
conditions 
(what trainee is 
given) 



Attainment 
standards 
(how well it is 
done) 



Use action verbs 

that are: 
Observance 
Measurable 
Verifiable 
Reliable (not 
prone to varying 
interpretation) 
Appropriate to 
task performance 



Examples of action 

verbs: 
operate, adjust, 
repair, remove, 
replace, multiply, 
label, record, file, 
measure, identify, 
compute, name, 
write, list and 
type 



Sample objective: 
Multiply 10 
decimal numbers, 
2 to 4 digits long, 
by a 1 -digit 
number and 
record the results 
of each 
calculation 



Inputs include: 
Tools and 
equipment 
References 
Job performance 
aids 
Special 
instructions 
Environment 
Problem 
situations or 
contingencies 



Using a hand-held 
calculator, given a 
sheet of problems 
and a pencil 



Criteria include: 
Accuracy 
requirements 
Qualitative 
requirements 
Time 

requirements 
Rate of 
production 
Amount of 
supervision 



Example of a 
standard: type- 
without error at 
a minimum speed 
of 40 words per 
minute 



With 100% 
accuracy, within 
3 minutes 



•Test items are written for learning objectives rather than from the sub- 
let matter as is the practice in some other instructional approaches. 



test is one in which a trainee's performance is compared 
with the average (norm) performance on the test. 
The nature of the (earning objective determines the type of 
criterion-test item needed, e.g. paper and pencil items are 
used to assess knowledge and performance items are us- 
ed to evaluate motor skills. Performance items are prefer- 
red, since they relate directly to job-task performance. Suf- 
ficient test items should be written for each learning objec- 
tive so that alternative forms, not just scrambled versions, 
of a test can be prepared, Alternative forms of a test are 
used by tainees who do not meet acceptable performance 
levels on their first test, are remedied and then retested. 

After the test items are written, those that were designed 
for loweNevel enabling objectives should be administered 
to a representative sample of the trainee population to 
measure entry-level behaviour which includes the 
knowledge and skills specific to the training programmes 
that incoming trainees already possess. 

Estimates of trainee capabilities must be verified or adjusted 
according to test results so that instruction meets trainees 
at the point of their need. When the test shows that some 
of the enabling objectives have already been acquired, they 
and their corresponding test items should be deleted' from 
the training programme. Whenever the test indicates that 
trainees lack prerequisite knowledge and skills, additional 
learning objectives and test items must be developed to fill 
the gap between previously made entry-level assumptions 
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and the true entry-level behaviour of trainees. Entry-level 
behaviour can also be determined by testing training pro- 
gramme participants just prior to the beginning of instruc- 
tion. Unfortunately, this approach leaves little time to make 
any modifications. 

The next step is to determine and specify how the terminal 
objectives and their respective enabling objectives will be 
sequenced and structured to provide for optimum learning. 
These decisions are made, as much as possible, on the basis 
of learning factors. However, given the inadequacy of 
knowledge about how different sequencing strategies af- 
fect learning and the difficulty of estimating transfer of lear- 
ning, they are based, in part, on judgment. 



Provide active participation 
and purposeful practice 



Objectives are first categorised as dependent, independent 
or supportive (those between which some transfer of lear- 
ning occurs). The dependent objectives are then sequenc- 
ed in accordance with the existing hierarchy of skills and 
knowledge so that prerequisite learning occurs first. Next, 
supporting objectives are placed as close to each other as 
possible without interfering with the dependent sequenc- 
ing. Independent objectives are sequenced last in accor- 
dance with such principles as simple to complex, familiar 
to unfamiliar, the order performed on the job, or any other 
order that is practical. Identical learning objectives are 
deleted and those with identicial actions are grouped 
together so that each is taught only once. 

To the extent that seems reasonable, the sequenced learn- 
ing objectives are grouped {structured) into units, blocks of 
instruction or other organised and manageable sets of ob- 
jectives in the way that they will be presented to trainees. 
Grouping is an arbitrary matter based on rational judgments 
regarding: (a) commonality of subject matter, (b) anticipated 
transfer of learning, (c) meaningful relationships, and (d) a 
natural beginning and ending point. 

Because test items measure specific learning objectives, 
they are structured at the same time as the learning objec- 
tives. This facilitates the grouping of test items into pre-tests, 
progress tests and post-tests. 

Develop 

The sequenced and structured learning objectives, from the 
previous phase— design-- identify what is to be taught. 
Once this is known, learning guidelines are considered in 
planning and developing the activities that will produce the 
desired learning outcomes. Learning guidelines are 
statements which specify teaching/learning principles, 
events and activities. Three basic guidelines (conditions) for 
learning are: 

(1) Provide guidance and prompts. Trainees should be 
shown and told how to do something. They need 
examples of what to do and explanations of how 



to do it. Performance ought to be as error-free as 
possible, since an error once made is likely to recur. 
Visual and verbal cues should be provided before 
and during practice to prevent guesswork and en 
sure correct performance[3]. 

(2) Provide feedback. Trainees need to know how they 
are doing. Such feedback should be prompt and 
indicate what is right or wrong and how to correct 
errors and improve performance. Knowledge and 
performance tests, as well as written work, should 
be used as teaching tools by fairly assessing the 
accomplishment of learning objectives and by pro- 
viding reinforcement for success and explanations 
for mistakes[4]. 

(3) Provide active participation and purposeful practice. 
Practising skills and reviewing knowledge is critical 
to learning and the retention of what is learned. 
While excessive practice becomes dull and is 
uneconomical, some degree of overlearning is 
desirable in order to enhance retention. Trainees 
learn by applying, in a meaningful way, what they 
are taught. Learning proceeds most effectively and 
tends to be most permanent when it is personally 
meaningful. Learning is an individual process in 
which trainees gain knowledge and skills and shape 
attitudes through their own activities, experiences 
and motivation. 

These guidelines serve as an aid in establishing appropriate 
learning activities specific to each objective. The learning 
activities then help identify what the instructor and/or the 
instructional materials must do to help the trainee attain the 
learning objective. 

Since there are numerous methods and various media to 
choose from, the instructional delivery system normally in- 
cludes a combination of appropriate teaching methods, the 
optional instructional media and a system of organising 
trainees and instructors to accomplish all learning objec- 
tives. Methods and media are selected on the basis of 
careful analysis of the instructional situation from several 
standpoints: (a) learning objective, (b) learning activities, (c) 
learning stage, (d) characteristics of the trainee population, 
(e) instructional staff, (f) equipment and facility constraints, 
(g) time and cost advantages, (h) whether group-paced or 
self-paced instruction is used, and (i) the extent to which 
instruction is individualised. An instructional management 
plan consists of the policies and procedures used in organis- 
ing, controlling and evaluating the delivery of instruction. 
This plan indicates: 

(a) How the training programme is to be conducted; 

(b) How the trainees are to be managed; 
(cl When and where they will be tested; 

Id) What the instructors and other support personnel 

are to do. and 
(e) How each of the many elements within the plan 
work together. 
Documents that are a part of the management plan specify 
the procedures to be used in scheduling, orienting incom- 
ing trainees, instructing, monitoring and evaluating both the 
trainees and instructors, and whatever else is needed to 
operate the training programme. 

Developing instructional materials is a time-consuming and 
expensive activity. It is essential, therefore, to search out and 
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consider the suitabilty of existing "off-the-shelf" items before 
committing resources to the development of new 
materials[5j. Candidate materials should be reviewed to 
determine their appropriateness to previously specified ob- 
jectives/ learning guidelines and activities, methods and 
media. Materials developed elsewhere that are nearly, but 
not entirely, adequate may be adapted for use. 

At this point the available existing instructional materials 
have been selected for use— some with minor revisions and 
some possibly with none. All that remains is to utilise the 
preceding analysis and planning to produce the necessary 
additional material*. 

The actual authoring process consists of writing or develop- 
ing many different types of instructional materials: (a) 
scripts, (b) sound/slide and videotape presentations, (c) pro- 
grammed texts, (d) instructional packages (modules), and 
(e) job performance aids. Initially, these materials should in- 
clude only the bare minimum of instruction, which will be 
augmented as needed for trainees to attain the learning ob- 
jectives. This "lean" strategy prevents the inclusion of non- 
essential information and instruction. 

All instructional materials should be reviewed by a 
knowledgeable individual as they become available. Then 
they are tried out on the more capable trainees with revi- 
sions and enrichment made as necessary. This assures the 
economies of minimal instruction and the correction of 
errors and deficiencies. Quantitative information about the 
effectiveness of instruction is obtained through validation 
procedures which provide further opportunities for revision. 

Validation procedures include both individual and group 
trials by selected members of the trainee population or 
representative learners. First, instructional materials are in- 
dividually tried out on two to five trainees, revised and tried 
again until areas of difficulty are corrected. Then group trials 
are conducted until it is proven that the trainees can attain 
the learning objectives as measured by criterion-referenced 
tests[6]. 

After validation, instructional materials are ready for quan- 
tity reproduction and implementation. When they are put 
into use before validation trials are completed, continual revi- 
sion, based on class experience, should be anticipated. 



Implement 

The implementation of the instructional management plan 
developed previously is the terminal step in planning and 
preparation and occurs before trainees arrive. Prior to the 
beginning of instruction, any needed refinements in the 
materials, procedures, tests and other components of the 
total training programmes are made. 

It is also necessary to make sure that the instructors and 
other personnel are trained in the techniques and pro- 
cedures with which they are not familiar. The emphasis is 
on assuring that enough qualified people are (a) at the right 
place at the right time, (b) that they know what they are 



* It is explicit in this approach that teaming objectives are prepared for every 
job-task selected for instruction and that these learning objectives are the 
basis for the selection and development of all instructional materials used 
in the training programme. Objectives are not derived from existing sub- 
ject matter as is the practice in some other approaches. 



to do, and (c) they have the materials, equipment, facilities 
and other resources necessary to do it. 

Complete training programme capabilty, consisting of lear- 
ning objectives, criterion-referenced tests, instructional 
materials, instructional management plan, a trained staff, 
and adequate facilities, equipment, tools and supplies 
should be available at this point. What remains to be done 
is to conduct instruction in accordance with the procedures 
and documentation contained in the instructional manage- 
ment plan. This is the "fruit of the labour". 



. . .necessary to make sure that 
instructors ... are trained in the 
techniques and procedures 



In addition to managing the resources, conducting instruc- 
tion, evaluating trainees and other activities, instructors need 
to keep records and collect data which is useful in evaluating 
the training programme. Among the records kept are: {a) 
test results and trainee performance for each learning ob- 
jective, (b) enrolment, (c) attendance, (d) attrition, and (e) 
scheduling data. 

A daily log is also maintained in which information such as 
the following is noted: 

(1) Instructor comments on trainees; 

(2) Time requirements for instruction and testing; 

(3) Problems with instructional materials; 

(4) Problems with people associated with instruction, 
and 

(5) Positive and negative incidents. 

All editorial, procedural and content changes that were 
made must also be documented in order to facilitate a com- 
plete and accurate programme evaluation. 

At the completion of each training cycle, the collected in- 
formation is made available as part of the internal evalua- 
tion in order to improve instruction for succeeding cycles. 



Control 

Despite the care taken in each procedural step while 
developing the training programme, it is still necessary to 
determine the effectiveness of the programme operating as 
a dynamic interrelated entity. The internal and external 
evaluation steps in this phase provide the information need- 
ed to determine if the training programme accomplished 
what it was designed and developed to do. 

Internal evaluation is based on information obtained within 
the training activity and is sometimes called a course review. 
It seeks to determine if instruction is providing trainees with 
the knowledge and skills necessary to attain the learning 
objectives. The principal measure of instructional effec- 
tiveness is the trainees' performance on criterion-referenced 
progress and post-tests. Other measures include: 
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(a) Instructor evaluations; 

(b) Trainee and instructor opinions concerning instruc- 
tion and testing, and 

(c) The amount of time required to complete each lear- 
ning objective (time on task). 

Training records and reports, as well as the notes kept by 
instructors during the conduct of instruction, provide addi- 
tional sources of valuable information on programme opera- 
tion and management. This is augmented by a check on 
the instructional materials, training facilities, equipment, 
tools and supplies to make sure they are adequate and 
appropriate. 



. . .most reliable. . .feedback is 
gained at the work site by 
observing the graduates' job 
performance. . . 



An examination of how the programme development effort 
was carried out is also part of the internal evaluation. The 
purpose is to identify any procedural omissions. Since the 
capabilities and performance of trainees and the manner 
in which instruction is conducted may change over time, 
internal evaluation is an ongoing process. 

The external evaluation is conducted to determine whether 
graduates are performing successfully on the job. It uses 
job performance as the standard to evaluate proficiency and 
the adequacy of the training programme. In a sense, this 
evaluation revalidates the job analysis and selection of tasks 
for instruction. 

External feedback is obtained by questionnaire, field inter- 
view or observation. The most reliable, but costly, feedback 
is gained at the work site by observing the graduates' job 
performance and by interviewing both graduates and their 
supervisors. Questionnaires completed by graduates and 
their immediate supervisors provide a valuable and less ex- 
pensive means for conducting field-based evaluation. 

The external evaluation should take place when the graduate 
has been on the job for one to three months. This timing 
is important because the graduate needs to get oriented 
and acquire a feel for what the job is all about. Further, the 
graduate and supervisor need time to formulate judgments 
concerning job preparation and performance. If more than 
three months lapse before the graduate and supervisor are 
contacted, it becomes difficult to discriminate between 
knowledge and skills acquired in training and those acquired 
on the job. As with internal evaluation, external evaluation 
is a recurring process which identifies problems and makes 
specific recommendations for appropriate adjustments. 

The final function is that of maintaining quality control of 
the instructional programme. This is done through revisions 
that keep training viable, relevant and up to data Revisions 
include changes in objectives, learning activities, realloca- 
tions of time and similar actions. They are made to improve 
trainee performance, reduce the rime required to complete 



instruction, or obtain the appropriate level of effectiveness 
at a lower cost. 

Since a considerable amount of effort has gone into the 
design and development of an instructional programme, revi- 
sions are approached with care. Decisions about what to 
revise should be based on the findings of internal and ex- 
ternal evaluations[7]. 

The evaluation and quality control steps complete the feed- 
back and interaction process, thereby forming the closed- 
loop instructional system shown in Figure 1. Continual 
evaluation and quality control are necessary to ensure that 
instruction is efficiently and effectively conducted to meet 
job requirements. 



Concluding Comments 

Training programmes based on nothing more than an in- 
structor's notion of what should be taught and how the 
training should be organised and delivered are in trouble. 
Such programmes will have to change in fundamental ways 
if they are to play a role in technological development and 
economic growth. 

The search for an approach with increased likelihood for 
providing training adequate to meet job-performance re- 
quirements has resulted in the advancement of an instruc- 
tional systems methodology. The iterative and derivative 
character of this methodology assures that instruction will 
meet job requirements when its procedures are conscien- 
tiously carried out. 

There is nothing really novel about these procedures. They 
are broadly accepted and have been used for a number of 
years. Nor is instructional systems a catchword to suggest 
something new; at most it is a process for doing something 
belter in deliberatly attempting to be systematic, analytic 
and comprehensive. 
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